Problem Statement¶
Context¶
Businesses like banks which provide service have to worry about problem of 'Customer Churn' i.e. customers leaving and joining another service provider. It is important to understand which aspects of the service influence a customer's decision in this regard. Management can concentrate efforts on improvement of service, keeping in mind these priorities.
Objective¶
You as a Data scientist with the bank need to build a neural network based classifier that can determine whether a customer will leave the bank or not in the next 6 months.
Data Dictionary¶
CustomerId: Unique ID which is assigned to each customer
Surname: Last name of the customer
CreditScore: It defines the credit history of the customer.
Geography: A customer’s location
Gender: It defines the Gender of the customer
Age: Age of the customer
Tenure: Number of years for which the customer has been with the bank
NumOfProducts: refers to the number of products that a customer has purchased through the bank.
Balance: Account balance
HasCrCard: It is a categorical variable which decides whether the customer has credit card or not.
EstimatedSalary: Estimated salary
isActiveMember: Is is a categorical variable which decides whether the customer is active member of the bank or not ( Active member in the sense, using bank products regularly, making transactions etc )
Exited : whether or not the customer left the bank within six month. It can take two values 0=No ( Customer did not leave the bank ) 1=Yes ( Customer left the bank )
In [ ]:
!pip install tensorflow
Importing necessary libraries¶
In [1]:
# Libraries to help with reading and manipulating data
import numpy as np
import pandas as pd
import matplotlib
import sklearn
# libaries to help with data visualization
import matplotlib.pyplot as pyplot
import matplotlib.pyplot as plt
import seaborn as sns
# Library to split data
from sklearn.model_selection import train_test_split
# library to import to standardize the data
from sklearn.preprocessing import StandardScaler
#Imports functions for evaluating the performance of machine learning models
from sklearn.metrics import confusion_matrix, f1_score,accuracy_score, recall_score, precision_score, classification_report
import keras
#Importing classback API
from keras import callbacks
# Importing tensorflow library
import tensorflow as tf
import time # Module for time-related operations.
# importing different functions to build models
from tensorflow.keras.layers import Dense, Dropout, InputLayer
from tensorflow.keras.models import Sequential
# Importing backend
from tensorflow.keras import backend
# Importing optimizers
from keras.optimizers import Adam
# Library to avoid the warnings
import warnings
warnings.filterwarnings("ignore")
Loading the dataset¶
In [2]:
#mounting Google Drive
from google.colab import drive
drive.mount('/content/drive')
In [3]:
#reading the dataset
df = pd.read_csv('/content/drive/MyDrive/Python/bank-1.csv')
In [4]:
#make a copy of data
data=df.copy()
Data Overview¶
In [5]:
#top 5 records of the data set
data.head()
Out[5]:
Observations
- In the above dataset, Exited is the target Variable with binay output values and this will be a Binary classification problem with two possible outcomes
In [6]:
#checking data info
data.info()
Observations
- There are 10000 rows and 14 columns in the data
- Out of 14 coulumns , 3 are Object type and 11 are numrical type
- There is no missing value in the data
In [7]:
#checking statistical summary of data
data.describe()
Out[7]:
Observations
- Credit Score ranges from 350 to 850, with 50% cutomers below score of 652
- Customers age ranges betweeen 18 to 92 years with 50% aging below 37 and 75% aging below 44
- Tenure ranges from 0-10 years, 50% below 5 years and 75% below 7 years
- Account Balance has a great variance from 0 to ~250K, average being ~76.5K , 50% customers having below ~97k and 75% below ~127K
- Estimated Salaray ranges from ~11$ to ~200K $, 50% maing below ~100K and 75% belo ~150K
- The Exited column data shows that 20.37 % of records are 1(Exited customers and 79.63 % customers are currently with the bank
In [8]:
#Check for data duplication
data.duplicated().sum()
Out[8]:
Observations
- There is no duplicated data
In [9]:
# checking the number of unique values in each column
data.nunique()
Out[9]:
Observations
- The columns Rownumber and CustomerId's have unique values which can be dropped as they do not add any value in EDA and model building
- Also the approximate 3000 values of Surnames will not add any value in EDA or Data Modelling even they they are not uniue, so we can drop it.
- Geography and Gender can be treated as categorical variables which we will label encode during Data Modelling
In [10]:
#check unqiue values of Variables
print(f"Unique values in 'Geography':",data['Geography'].unique())
print(f"Unique values in 'Gender':",data['Gender'].unique())
print(f"Unique values in 'Tenure':",data['Tenure'].unique())
print(f"Unique values in 'No of Products':",df['NumOfProducts'].unique())
Observations
- From above we can infer the following variables as
- Numerical(6) - CreditScore,Age,Tenure,Balance,EstimatedSalary,Exited,NoofProd
- Categorical (4) - Gender,Geography,hasCard,IaActivemem,
- Object(1) - Surname
Exploratory Data Analysis¶
Data Cleaning¶
In [11]:
#Dropping rownumber and customerid before EDA due to uniqueness in each row
# enrollee_id is unique for each candidate and might not add value to modeling
data.drop(["RowNumber","CustomerId","Surname"],axis=1, inplace=True)
#validate the dropped columns
data.info()
In [12]:
data.head()
Out[12]:
Function Definitions for EDA.¶
In [ ]:
# function to plot a boxplot and a histogram along the same scale.
def histogram_boxplot(data, feature, figsize=(12, 7), kde=False, bins=None):
"""
Boxplot and histogram combined
data: dataframe
feature: dataframe column
figsize: size of figure (default (12,7))
kde: whether to the show density curve (default False)
bins: number of bins for histogram (default None)
"""
f2, (ax_box2, ax_hist2) = plt.subplots(
nrows=2, # Number of rows of the subplot grid= 2
sharex=True, # x-axis will be shared among all subplots
gridspec_kw={"height_ratios": (0.25, 0.75)},
figsize=figsize,
) # creating the 2 subplots
sns.boxplot(
data=data, x=feature, ax=ax_box2, showmeans=True, color="violet"
) # boxplot will be created and a triangle will indicate the mean value of the column
sns.histplot(
data=data, x=feature, kde=kde, ax=ax_hist2, bins=bins, palette="winter"
) if bins else sns.histplot(
data=data, x=feature, kde=kde, ax=ax_hist2
) # For histogram
ax_hist2.axvline(
data[feature].mean(), color="green", linestyle="--"
) # Add mean to the histogram
ax_hist2.axvline(
data[feature].median(), color="black", linestyle="-"
) # Add median to the histogram
In [ ]:
# function to create labeled barplots
def labeled_barplot(data, feature, perc=False, n=None):
"""
Barplot with percentage at the top
data: dataframe
feature: dataframe column
perc: whether to display percentages instead of count (default is False)
n: displays the top n category levels (default is None, i.e., display all levels)
"""
total = len(data[feature]) # length of the column
count = data[feature].nunique()
if n is None:
plt.figure(figsize=(count + 1, 5))
else:
plt.figure(figsize=(n + 1, 5))
plt.xticks(rotation=90, fontsize=15)
ax = sns.countplot(
data=data,
x=feature,
palette="Paired",
order=data[feature].value_counts().index[:n].sort_values(),
)
for p in ax.patches:
if perc == True:
label = "{:.1f}%".format(
100 * p.get_height() / total
) # percentage of each class of the category
else:
label = p.get_height() # count of each level of the category
x = p.get_x() + p.get_width() / 2 # width of the plot
y = p.get_height() # height of the plot
ax.annotate(
label,
(x, y),
ha="center",
va="center",
size=12,
xytext=(0, 5),
textcoords="offset points",
) # annotate the percentage
plt.show() # show the plot
In [ ]:
# function to plot stacked bar chart
def stacked_barplot(data, predictor, target):
"""
Print the category counts and plot a stacked bar chart
data: dataframe
predictor: independent variable
target: target variable
"""
count = data[predictor].nunique()
sorter = data[target].value_counts().index[-1]
tab1 = pd.crosstab(data[predictor], data[target], margins=True).sort_values(
by=sorter, ascending=False
)
print(tab1)
print("-" * 120)
tab = pd.crosstab(data[predictor], data[target], normalize="index").sort_values(
by=sorter, ascending=False
)
tab.plot(kind="bar", stacked=True, figsize=(count + 1, 5))
plt.legend(
loc="lower left", frameon=False,
)
plt.legend(loc="upper left", bbox_to_anchor=(1, 1))
plt.show()
In [ ]:
### Function to plot distributions
def distribution_plot_wrt_target(data, predictor, target):
fig, axs = plt.subplots(2, 2, figsize=(12, 10))
target_uniq = data[target].unique()
axs[0, 0].set_title("Distribution of target for target=" + str(target_uniq[0]))
sns.histplot(
data=data[data[target] == target_uniq[0]],
x=predictor,
kde=True,
ax=axs[0, 0],
color="teal",
)
axs[0, 1].set_title("Distribution of target for target=" + str(target_uniq[1]))
sns.histplot(
data=data[data[target] == target_uniq[1]],
x=predictor,
kde=True,
ax=axs[0, 1],
color="orange",
)
axs[1, 0].set_title("Boxplot w.r.t target")
sns.boxplot(data=data, x=target, y=predictor, ax=axs[1, 0], palette="gist_rainbow")
axs[1, 1].set_title("Boxplot (without outliers) w.r.t target")
sns.boxplot(
data=data,
x=target,
y=predictor,
ax=axs[1, 1],
showfliers=False,
palette="gist_rainbow",
)
plt.tight_layout()
plt.show()
Univariate Analysis¶
Observation:CreditScore¶
In [ ]:
#Plot CreditScore
histogram_boxplot(data, "CreditScore")
Obeservations
- CreditScoe is a slightly left skwerd distribution,median around 650 and some outliers below 400, and Q1-Q3 ranges 584 to 718
Observation:Age¶
In [ ]:
#Plot Age
histogram_boxplot(data, "Age")
Observations
- Age is a right skewed distribution,customer density below age 50 is high and tapering after 60. There are some outliers above 60
Observation:Tenure¶
In [ ]:
#Plot Tenure
histogram_boxplot(data, "Tenure")
Observations
- Customer across tenures are somewhat evently distributed between 1-9 years(~1000) ,however customers with 0 or 10 years tenure are almost half
Observation:Balance¶
In [ ]:
#Plot Balance
histogram_boxplot(data, "Balance")
Observations
- Approximately 3500 customers have zero balance
- Apart from zero balance, the rest of the distribution is normal
Observation: EstimatedSalary¶
In [ ]:
#Plot Estimated Salary
histogram_boxplot(data, "EstimatedSalary")
Observations
- Esitmated Salary is evenly distributed
Observation on Gender¶
In [ ]:
labeled_barplot(data, "Gender",perc=True)
Observations
- There are 54.6% male and 45.4% females in the data set
Observation on Geography¶
In [ ]:
labeled_barplot(data, "Geography",perc=True)
Observations
- Half customers in Framce and rest 25% each are from Germany and Spain
Observation on HasCrard¶
In [ ]:
labeled_barplot(data, "HasCrCard",perc=True)
Observations
- Aproximately 30% customers do not have card and 70% have cards
Observation on IsActiveMember¶
In [ ]:
labeled_barplot(data, "IsActiveMember",perc=True)
Observation on NumOfProducts¶
In [ ]:
labeled_barplot(data, "NumOfProducts",perc=True)
Observations
- Approximately 51% of customers have 1 product and 46%
- There are very few(2.7%) customers with 3 products and very few(0.6%)with 4 products
Bivariate Analysis¶
Pairplot¶
In [ ]:
sns.pairplot(data, hue ='Exited' , diag_kind='hist')
Out[ ]:
Observations
- Most customers who exited are between 40-62 age .Also customers above 40 who are not active members have higher churn
- Customers below credit score 400 have a high density of leaving
- All customers with 4 produts and most with 3 products have left the bank
- Non active members with 1 product tend and active members with 2 products tend to stay
HeatMap¶
In [ ]:
plt.figure(figsize=(10,8))
sns.heatmap(data.corr(numeric_only = True), annot=True, vmin=-1, vmax=1, fmt=".2f", cmap="coolwarm",linewidths=0.5)
plt.show()
Observations There are only two noticable correlations :
- No of Products and Balance are negatively co-related with a factor of 3 (-ie: when no of products increase, Balance decreases)
- Target Variable is postively co-related with Age with a factor of 0.29
Gender wrt Target¶
In [ ]:
stacked_barplot(df,"Gender","Exited")
observations
- ~33% of female customers left and ~20% of male customers left
Geography wrt Target¶
In [ ]:
stacked_barplot(df,"Geography","Exited")
Observations
- Germany has 32% customers leaving,France 16% and Spain too 16%, Germany has almost double percentage of customers leaving than other two countries
HasCrCard wrt Target¶
In [ ]:
stacked_barplot(df,"HasCrCard","Exited")
Observations
- Having Card is not a single factor influencing customers leaving the bank as both customers having and not having Card have left the bank in equal ratio.
IsAtiveMember wrt Target¶
In [ ]:
stacked_barplot(df,"IsActiveMember","Exited")
Observations
- 27% of Active members left the bank wheras 14% of customers no active members have left the bank
No of Products Wrt Target¶
In [ ]:
stacked_barplot(df,"NumOfProducts","Exited")
Observations
- cusometrs with 3,4 products have a less and they have a high rate of leaving the bank
- Most customers with 2 produts stay.
Age Distribution wrt Target¶
In [ ]:
distribution_plot_wrt_target(data, "Age", "Exited")
Observations
- Most of customers who did not leave the bank were between ~32 to ~42 age and those who left were bettween ~37 to ~52 Age
Credit Score wrt Target¶
In [ ]:
distribution_plot_wrt_target(data, "CreditScore", "Exited")
Observations
- Credit score does not have a direct realtion with cusomers leaving, however there are customers with very low credit scores who have left the bank
Balance wrt Target¶
In [ ]:
distribution_plot_wrt_target(data, "Balance", "Exited")
Observation
- Customers Leaving are high b/w 50000 to 170000 Balance
EstimatedSalary wrt Target¶
In [ ]:
distribution_plot_wrt_target(data, "EstimatedSalary", "Exited")
Observations
- No specific outcomes on existing customers across Estimated Salary Distribution
Tenure wrt Target¶
In [ ]:
distribution_plot_wrt_target(data, "Tenure", "Exited")
Observation
- Most Cusomerts who have left the bank have tenures from 2-8 years
No Of Products wrt Target¶
In [ ]:
distribution_plot_wrt_target(data, "NumOfProducts", "Exited")
EDA OBSERVATIONS-CONSOLIDATED¶
After doing univariate and Bivariate analysis, the significant points are
a. Customers in the age range of 40-62 tend to leave the bank, especially those who are not active members.
b. 50% customers are in France, however customers churn in Germany is more (approximately 33% of its customer base)
c. Other significant factors which influnce customeres leaving the bank are :
- higher no of products( 3,4) : leave are their % is in total is 3%
- Female members tend to leave more than male (33% vs 20%)
- Customers with low credit scores(below 400)
- Non Active member with more than 1 product tends to leave
Data Preprocessing¶
Steps Relevant in Data Proprocessing befoe model building¶
-Dropping of Columns which whi not add value (Customerid,RowId,Surname)-Accomplished before EDA
- Encoding of Categorical Variables to numerical Values : Georgraphy and Gender
- Splitting Data in Train and Test sets
- Normalizing and Sacling Numerical Variables CreditScore, Age, Tenure, Balance, EstimatedSalary
In [13]:
#Step1: Keep a copy of Data before pre-processing
data1=data.copy()
Train-validation-test Split¶
In [14]:
#Splitting the target Variables and Predictors
X = data1.drop(["Exited"], axis=1)
Y = data1["Exited"]
In [15]:
# splitting the data in 80:20 ratio for train and temporary data
X_train, X_temp, Y_train, Y_temp = train_test_split(X, Y, test_size=0.2,random_state=1)
In [16]:
# splitting the temporary data in 50:50 ratio for validation and test data
X_val,X_test,Y_val,Y_test = train_test_split(X_temp,Y_temp,test_size=0.5,random_state=1)
In [17]:
print("Number of rows in train data =", X_train.shape[0])
print("Number of rows in validation data =", X_val.shape[0])
print("Number of rows in test data =", X_test.shape[0])
In [18]:
print("Number of col in train data =", X_train.shape[1])
print("Number of col in validation data =", X_val.shape[1])
print("Number of col in test data =", X_test.shape[1])
Dummy Variable Creation¶
In [19]:
# List of columns to be converted into dummy variables
categorical_columns = ['Gender', 'Geography', 'HasCrCard', 'IsActiveMember']
In [20]:
X_train = pd.get_dummies(X_train,columns=categorical_columns, drop_first=True)
X_train = X_train.astype(float)
X_val = pd.get_dummies(X_val,columns=categorical_columns, drop_first=True)
X_val = X_val.astype(float)
X_test = pd.get_dummies(X_test,columns=categorical_columns, drop_first=True)
X_test = X_test.astype(float)
print(X_train.shape, X_val.shape, X_test.shape)
In [21]:
# Calculate and print target distribution in each set
def print_distribution(y, set_name):
print(f"{set_name} Distribution:")
print(Y.value_counts(normalize=True) * 100)
print()
print_distribution(Y_train, "Training Set")
print_distribution(Y_val, "Validation Set")
print_distribution(Y_test, "Test Set")
Data Normalization¶
In [22]:
scaler = StandardScaler()
# Here, we are passing all the features (numerical and categorical), that's okay as min-max scaler will not change values of categorical variables
X_train_normalized = scaler.fit_transform(X_train)
In [23]:
X_val_normalized = scaler.transform(X_val)
In [24]:
X_test_normalized = scaler.transform(X_test)
In [25]:
X_val_normalized.shape
Out[25]:
Model Building¶
Model Evaluation Criterion¶
Write down the logic for choosing the metric that would be the best metric for this business scenario.
Accuracy followed by F1 Score will be the most significant criterions for evaluation based on the logic outlined below:
-- Accurancy Since the data set is imbalanced(the no of target customers who are leaving are significantly lesser than the customers who are existing),high accuracy ie correct predictions will be the most important metric
-- Prescision, Recall and F1 Score
- Maximise prescision : by ensuring the predicted customers leaving the bank are actually leaving
- Maximize Recall : Ensuring customers predicting actual customers leaving and minimizing False Negatives (scenario where cutomers are wrongly predicted to curn where they do not leave the bank)
- F1 Score: This will give a single metric which will capture both Prescion and Recall Criteria mentioned above especially important in a imbalanced class distribution
Utility Functions¶
In [26]:
def plot(history, name):
"""
Function to plot loss/accuracy
history: an object which stores the metrics and losses.
name: can be one of Loss or Accuracy
"""
fig, ax = plt.subplots() #Creating a subplot with figure and axes.
plt.plot(history.history[name]) #Plotting the train accuracy or train loss
plt.plot(history.history['val_'+name]) #Plotting the validation accuracy or validation loss
plt.title('Model ' + name.capitalize()) #Defining the title of the plot.
plt.ylabel(name.capitalize()) #Capitalizing the first letter.
plt.xlabel('Epoch') #Defining the label for the x-axis.
fig.legend(['Train', 'Validation'], loc="outside right upper") #Defining the legend, loc controls the position of the legend.
In [27]:
# defining a function to compute different metrics to check performance of a classification model built using statsmodels
def model_performance_classification(
model, predictors, target, threshold=0.5
):
"""
Function to compute different metrics to check classification model performance
model: classifier
predictors: independent variables
target: dependent variable
threshold: threshold for classifying the observation as class 1
"""
# checking which probabilities are greater than threshold
pred = model.predict(predictors) > threshold
# pred_temp = model.predict(predictors) > threshold
# # rounding off the above values to get classes
# pred = np.round(pred_temp)
acc = accuracy_score(target, pred) # to compute Accuracy
recall = recall_score(target, pred, average='weighted') # to compute Recall
precision = precision_score(target, pred, average='weighted') # to compute Precision
f1 = f1_score(target, pred, average='weighted') # to compute F1-score
# creating a dataframe of metrics
df_perf = pd.DataFrame(
{"Accuracy": acc, "Recall": recall, "Precision": precision, "F1 Score": f1,},
index=[0],
)
return df_perf
In [28]:
# defining the batch size and # epochs upfront as we'll be using the same values for all models
epochs = 50
batch_size = 64
Neural Network with SGD Optimizer¶
Since this is a binary classification problem , we will build a simple Feed Forward Neural Network
- Let's start with a neural network consisting of
- two hidden layers with 14 and 7 neurons respectively
- activation function of ReLU.
- SGD as the optimizer
In [29]:
# clears the current Keras session, resetting all layers and models previously created, freeing up memory and resources.
tf.keras.backend.clear_session()
In [30]:
#Initializing the neural network
model = Sequential()
model.add(Dense(14,activation="relu",input_dim=X_train_normalized.shape[1]))
#model.add(Dense(7,activation="relu"))
model.add(Dense(7,activation="relu"))
model.add(Dense(1,activation="sigmoid"))
In [31]:
model.summary()
In [32]:
optimizer = tf.keras.optimizers.SGD() # defining SGD as the optimizer to be used
model.compile(loss='binary_crossentropy', optimizer=optimizer,metrics=['accuracy','f1_score'])
In [33]:
start = time.time()
history = model.fit(X_train_normalized, Y_train, validation_data=(X_val_normalized,Y_val) , batch_size=batch_size, epochs=epochs)
end=time.time()
In [34]:
print("Time taken in seconds ",end-start)
In [35]:
plot(history,'loss')
In [36]:
plot(history,'accuracy')
In [37]:
model_0_train_perf = model_performance_classification(model, X_train_normalized, Y_train)
model_0_train_perf
Out[37]:
In [38]:
model_0_val_perf = model_performance_classification(model, X_val_normalized, Y_val)
model_0_val_perf
Out[38]:
Comments on Model Performance
- Training and Validation Accurancy increased over epochs indication that model generalised and learn well to validation set
- Training and Validation loss decreased steadity showing that model converged well
- Validation F1-score remained constant at .3036 across epochs showing model is failing to identify true positive and reducing false negatives
- The performance metrics (accuracy, recall, precision, F1 score) showing that the model is generalising well and not overfitting ,It is showing strong performance.
Observations
Neural Network with Adam Optimizer-M1¶
In [39]:
# clears the current Keras session, resetting all layers and models previously created, freeing up memory and resources.
tf.keras.backend.clear_session()
In [40]:
#Initializing the neural network
model_1 = Sequential()
# Input layer and the first hidden layer
model_1.add(Dense(14,activation="relu",input_dim = X_train_normalized.shape[1]))
# Second hidden layer
model_1.add(Dense(14, activation="relu"))
# third hidden layer
model_1.add(Dense(7, activation="relu"))
# output layer
model_1.add(Dense(1, activation="sigmoid"))
In [41]:
# Compile the model with Adam optimizer and a specified learning rate
#adam = Adam(learning_rate=0.001)
optimizer=keras.optimizers.Adam(learning_rate=0.001)
model_1.compile(optimizer=optimizer, loss='binary_crossentropy', metrics=['accuracy','f1_score'])
In [42]:
start = time.time()
history = model_1.fit(X_train_normalized, Y_train, validation_data=(X_val_normalized,Y_val) , batch_size=batch_size, epochs=epochs)
end=time.time()
In [43]:
print("Time taken in seconds ",end-start)
In [44]:
plot(history,'loss')
In [45]:
plot(history,'accuracy')
In [46]:
model_1_train_perf = model_performance_classification(model_1, X_train_normalized, Y_train)
model_1_train_perf
Out[46]:
In [47]:
model_1_val_perf = model_performance_classification(model_1, X_val_normalized, Y_val)
model_1_val_perf
Out[47]:
Comments on Model Performance
- The validation loss is consistently lesser than the training loss & Validation accuracy consistently hight than the Training accuracy which indicate good generalisation but could also be data leakage.
- Training and Validation loss decreased steadity showing that model converged well
- Validation F1-score remained constant at .3036 across epochs showing model is failing to identify true positive and reducing false negatives
- The Model shows high accurancy on train & val sets,Good Precision and recall and cosistent f1 scores
Neural Network with Adam Optimizer and Dropout-M2¶
In [48]:
# clears the current Keras session, resetting all layers and models previously created, freeing up memory and resources.
tf.keras.backend.clear_session()
In [49]:
#Initializing the neural network
model_2 = Sequential()
# Input layer and the first hidden layer
model_2.add(Dense(14,activation="relu",input_dim = X_train_normalized.shape[1]))
model_2.add(Dropout(0.5))
# Second hidden layer
model_2.add(Dense(14, activation="relu"))
model_2.add(Dropout(0.5))
# third hidden layer
model_2.add(Dense(7, activation="relu"))
model_2.add(Dropout(0.5))
# output layer
model_2.add(Dense(1, activation="sigmoid"))
In [50]:
# Compile the model with Adam optimizer and a specified learning rate
#adam = Adam(learning_rate=0.001)
optimizer=keras.optimizers.Adam(learning_rate=0.001)
model_2.compile(optimizer=optimizer, loss='binary_crossentropy', metrics=['accuracy','f1_score'])
In [51]:
start = time.time()
history = model_2.fit(X_train_normalized, Y_train, validation_data=(X_val_normalized,Y_val) , batch_size=batch_size, epochs=epochs)
end=time.time()
In [52]:
print("Time taken in seconds ",end-start)
In [53]:
plot(history,'loss')
In [54]:
plot(history,'accuracy')
In [55]:
model_2_train_perf = model_performance_classification(model_2, X_train_normalized, Y_train)
model_2_train_perf
Out[55]:
In [56]:
model_2_val_perf = model_performance_classification(model_2, X_val_normalized, Y_val)
model_2_val_perf
Out[56]:
Comments on Model Performance
- Validation accuracy is consistently over traning accuracy which could be an issue.
- Validation loss is consistently below training loss which seems unusual
- However, model seems to converge well and genaralise well but indicate some data leakage or other possible issue
- Validation F1-score remained constant at .3036 across epochs showing model is failing to identify true positive and reducing false negatives
- The model shows strong performance with high accuracy and recall, which is important for identifying customers likely to churn. However, there scope to improve precision to reduce false positives
Neural Network with Balanced Data (by applying SMOTE) and SGD Optimizer-M3¶
In [57]:
# clears the current Keras session, resetting all layers and models previously created, freeing up memory and resources.
tf.keras.backend.clear_session()
In [58]:
# To oversample and undersample data
from imblearn.over_sampling import SMOTE
In [59]:
print("Before Oversampling, counts of label 'Existing Customer': {}".format(sum(Y_train == 1)))
print("Before Oversampling, counts of label 'Churned Customer': {} \n".format(sum(Y_train == 0)))
print("Before Oversampling, counts of label 'Existing Customer-val': {}".format(sum(Y_val == 1)))
print("Before Oversampling, counts of label 'Churned Customer-val': {} \n".format(sum(Y_val == 0)))
print("Before Oversampling, counts of label 'Existing Customer-test': {}".format(sum(Y_test == 1)))
print("Before Oversampling, counts of label 'Churned Customer-test': {} \n".format(sum(Y_test == 0)))
In [60]:
# Synthetic Minority Over Sampling Technique
sm = SMOTE(sampling_strategy=1, k_neighbors=5, random_state=1)
X_train_over, y_train_over = sm.fit_resample(X_train, Y_train)
In [61]:
print("After Oversampling, counts of label 'Attrited Customer': {}".format(sum(y_train_over == 1)))
print("After Oversampling, counts of label 'Existing Customer': {} \n".format(sum(y_train_over == 0)))
In [62]:
print("After Oversampling, the shape of train_X: {}".format(X_train_over.shape))
print("After Oversampling, the shape of train_y: {} \n".format(y_train_over.shape))
In [63]:
# Normalize the features (do this AFTER SMOTE)
scaler = StandardScaler()
X_train_sm_normalized = scaler.fit_transform(X_train_over)
X_val_sm_normalized = scaler.transform(X_val)
X_test_sm_normalized = scaler.transform(X_test)
In [64]:
#Initializing the neural network
model_3= Sequential()
#Input layer and first hidden layer
model_3.add(Dense(14,activation="relu",input_dim=X_train_sm_normalized.shape[1]))
#Second hidden layer
model_3.add(Dense(14, activation="relu"))
#Third hidden layer
model_3.add(Dense(7,activation="relu"))
#Output layer
model_3.add(Dense(1,activation="sigmoid"))
In [65]:
optimizer = tf.keras.optimizers.SGD() # defining SGD as the optimizer to be used
model_3.compile(loss='binary_crossentropy', optimizer=optimizer,metrics=['accuracy','f1_score'])
In [66]:
start = time.time()
history = model_3.fit(X_train_sm_normalized, y_train_over, validation_data=(X_val_sm_normalized,Y_val) , batch_size=batch_size, epochs=epochs)
end=time.time()
In [67]:
print("Time taken in seconds ",end-start)
In [68]:
plot(history,'loss')
In [69]:
plot(history,'accuracy')
In [70]:
model_3_train_perf = model_performance_classification(model_3, X_train_sm_normalized, y_train_over)
model_3_train_perf
Out[70]:
In [71]:
model_3_val_perf = model_performance_classification(model_3, X_val_sm_normalized, Y_val)
model_3_val_perf
Out[71]:
Comments on Model Performance
- Training accuracy consistently increased over epochs showing model is learning weel from data.Validation accouracy shows good genaralisation *Training and Validation loss decrease over time
- However, model seems to converge well and genaralise well but indicate some data leakage or other possible issue
- The training F1 score is consistently around 66.34%, whereas the validation F1 score remains constant at 38.36%. The low validation F1 score compared to the training F1 score indicates that the model might not be performing as well in identifying true positives on the validation set. *Model is genalising and performing well, both recall and precision are close on training and validation sets.
Neural Network with Balanced Data (by applying SMOTE) and Adam Optimizer-M4¶
In [72]:
# clears the current Keras session, resetting all layers and models previously created, freeing up memory and resources.
tf.keras.backend.clear_session()
In [73]:
#Initializing the neural network
model_4 = Sequential()
# Input layer and the first hidden layer
model_4.add(Dense(14,activation="relu",input_dim = X_train_sm_normalized.shape[1]))
# Second hidden layer
model_4.add(Dense(14, activation="relu"))
# third hidden layer
model_4.add(Dense(7, activation="relu"))
# output layer
model_4.add(Dense(1, activation="sigmoid"))
In [74]:
# Compile the model with Adam optimizer and a specified learning rate
#adam = Adam(learning_rate=0.001)
optimizer=keras.optimizers.Adam(learning_rate=0.001)
model_4.compile(optimizer=optimizer, loss='binary_crossentropy', metrics=['accuracy','f1_score'])
In [75]:
start = time.time()
history = model_4.fit(X_train_sm_normalized, y_train_over, validation_data=(X_val_sm_normalized,Y_val) , batch_size=batch_size, epochs=epochs)
end=time.time()
In [76]:
print("Time taken in seconds ",end-start)
In [77]:
plot(history,'loss')
In [78]:
plot(history,'accuracy')
In [79]:
model_4_train_perf = model_performance_classification(model_4, X_train_sm_normalized, y_train_over)
model_4_train_perf
Out[79]:
In [80]:
model_4_valid_perf = model_performance_classification(model_4, X_val_sm_normalized, Y_val)
model_4_valid_perf
Out[80]:
Comments on Model Performance
- Training Accuracy shows good performncae,validation accuracy shows good genalisations but some fluctuations which stablisizes well.
- Both training and validation losses are decreasing, indicating that the model is learning and not overfitting.
- Both training and validation accuracy are increasing, suggesting the model is improving its performance on both seen and unseen data.
- The training F1 score is consistently around 66.34%, whereas the validation F1 score remains constant at 38.36%. The low validation F1 score compared to the training F1 score indicates that the model might not be performing as well in identifying true positives on the validation set. *The model is performing well, with high and consistent metrics across both training and validation sets.It shows high accuracy, balanced precision and recall and consistent f1 scores
Neural Network with Balanced Data (by applying SMOTE), Adam Optimizer, and Dropout-M5¶
In [81]:
# clears the current Keras session, resetting all layers and models previously created, freeing up memory and resources.
tf.keras.backend.clear_session()
In [82]:
#Initializing the neural network
model_5 = Sequential()
# Input layer and the first hidden layer
model_5.add(Dense(14,activation="relu",input_dim = X_train_sm_normalized.shape[1]))
model_5.add(Dropout(0.5))
# Second hidden layer
model_5.add(Dense(14, activation="relu"))
model_5.add(Dropout(0.5))
# third hidden layer
model_5.add(Dense(7, activation="relu"))
model_5.add(Dropout(0.5))
# output layer
model_5.add(Dense(1, activation="sigmoid"))
In [83]:
# Compile the model with Adam optimizer and a specified learning rate
#adam = Adam(learning_rate=0.001)
optimizer=keras.optimizers.Adam(learning_rate=0.001)
model_5.compile(optimizer=optimizer, loss='binary_crossentropy', metrics=['accuracy','f1_score'])
In [84]:
start = time.time()
history = model_5.fit(X_train_sm_normalized, y_train_over, validation_data=(X_val_sm_normalized,Y_val) , batch_size=batch_size, epochs=epochs)
end=time.time()
In [85]:
print("Time taken in seconds ",end-start)
In [86]:
plot(history,'loss')
In [87]:
plot(history,'accuracy')
In [88]:
model_5_train_perf = model_performance_classification(model_5, X_train_sm_normalized, y_train_over)
model_5_train_perf
Out[88]:
In [89]:
model_5_valid_perf = model_performance_classification(model_5, X_val_sm_normalized, Y_val)
model_5_valid_perf
Out[89]:
Comments on Model Performance
*We see decrease in both training and bvalidation loss over epochs and increase in training and validation accuracy over times
- Model is not overfitting , there are minor fluctuations
- Model shows good genarisations,consistent f1 scores (suggesting stable balance between recall and precision)
Model Performance Comparison and Final Model Selection¶
In [90]:
# training performance comparison
models_train_comp_df = pd.concat(
[
model_0_train_perf.T,
model_1_train_perf.T,
model_2_train_perf.T,
model_3_train_perf.T,
model_4_train_perf.T,
model_5_train_perf.T,
],
axis=1,
)
models_train_comp_df.columns = [
"NN with SGD Optimizer-M0",
"NN with Adam Optimizer-M1",
"NNk with Adam Optimizer and Dropout-M2",
"NN(SMOTE) and SGD Optimizer-M3",
"NN( SMOTE) and Adam Optimizer-M4",
"NN(SMOTE), Adam Optimizer, and Dropout-M5"
]
In [91]:
#Validation performance comparison
models_valid_comp_df = pd.concat(
[
model_0_val_perf.T,
model_1_val_perf.T,
model_2_val_perf.T,
model_3_val_perf.T,
model_4_valid_perf.T,
model_5_valid_perf.T,
],
axis=1,
)
models_valid_comp_df.columns = [
"NN with SGD Optimizer-M0",
"NN with Adam Optimizer-M1",
"NNk with Adam Optimizer and Dropout-M2",
"NN(SMOTE) and SGD Optimizer-M3",
"NN( SMOTE) and Adam Optimizer-M4",
"NN(SMOTE), Adam Optimizer, and Dropout-M5"
]
In [92]:
models_train_comp_df
Out[92]:
In [93]:
models_valid_comp_df
Out[93]:
Final Model Selection¶
- On comparing all the model performances listed above , Adam Optimizer-M4 NN(SMOTE) is the best model and will be the Final model for selction .
Adam Optimizer-M4 NN(SMOTE)
Training Metrics: Accuracy: 87.70%, Recall: 87.7%, Precision: 88.24%, F1 Score: 87.73%
Validation Metrics: Accuracy: 87.1.00%, Recall: 87.70%, Precision: 86.52%, F1 Score: 86.74%
Reasons:
- Highest accuracy and F1 Score(indicating good overall performance and balance between precision and recall.)
- Validation set generasing best showing no signs of overfitting
Final Model¶
In [95]:
# clears the current Keras session, resetting all layers and models previously created, freeing up memory and resources.
tf.keras.backend.clear_session()
In [96]:
#Initializing the neural network
model_test = Sequential()
# Input layer and the first hidden layer
model_test.add(Dense(14,activation="relu",input_dim = X_test_normalized.shape[1]))
# Second hidden layer
model_test.add(Dense(14, activation="relu"))
# third hidden layer
model_test.add(Dense(7, activation="relu"))
# output layer
model_test.add(Dense(1, activation="sigmoid"))
In [97]:
# Compile the model with Adam optimizer and a specified learning rate
#adam = Adam(learning_rate=0.001)
optimizer=keras.optimizers.Adam(learning_rate=0.001)
model_test.compile(optimizer=optimizer, loss='binary_crossentropy', metrics=['accuracy','f1_score'])
In [98]:
start = time.time()
history = model_test.fit(X_train_sm_normalized, y_train_over, validation_data=(X_test_sm_normalized,Y_test) , batch_size=batch_size, epochs=epochs)
end=time.time()
In [99]:
model_test_perf = model_performance_classification(model_test, X_test_sm_normalized, Y_test)
model_test_perf
Out[99]:
Actionable Insights and Business Recommendations¶
Actionable Insighst
The models shows high accurancy, recall, presicion and f1_score good generalisation showing that it can reliably predict customers whih will leave the bank.
Banks should leverage important features as Balance,CreditScore, Active Membership, Geography to understand reasons for customers leaving the bank.
Further improvements in the model can be achieved using more hypertuning paramaters tweaking like Requlaisation,Early Stopping , using class weights
Business Recommendations
Using the insights from the model, bank should target at-risk customers with tailored engagement strategies , ex: loyalty rewards, exclusive discounts , financial advise
AT-risk customers should be provided dedicated customer support with bank offerings and resolving their issues
Since bank provides services, using predictive insights the business team should analyse at-risk customers feeback and usage pattern and identify areas of improvements :ex: improving mobile banking features, making loan applicationn process easy
Bank can offer Targeted marketing Campaingns based on churn insights focusing on features used by at-risk customers
Tailored Strategties for specific groups like High Value customers or long termn customers or new customers who are at churn risk will also help bank retain customers
Power Ahead